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Abstract 

Max weighted queue (MWQ) control policy is a widely used cross-layer control policy that 
achieves queue stability and a reasonable delay performance. In most of the existing literature, it is 
assumed that optimal MWQ policy can be obtained instantaneously at every time slot. However, this 
assumption may be unrealistic in time varying wireless systems, especially when there is no closed- 
form MWQ solution and iterative algorithms have to be applied to obtain the optimal solution. This 
paper investigates the convergence behavior and the queue delay performance of the conventional 
MWQ iterations in which the channel state information (CSI) and queue state information (QSI) are 
changing in a similar timescale as the algorithm iterations. Our results are established by studying 
the stochastic stability of an equivalent virtual stochastic dynamic system (VSDS), and an extended 
Foster-Lyapunov criteria is applied for the stability analysis. We derive a closed form delay bound of 
the wireless network in terms of the CSI fading rate and the sensitivity of MWQ policy over CSI and 
QSI. Based on the equivalent VSDS, we propose a novel MWQ iterative algorithm with compensation 
to improve the tracking performance. We demonstrate that under some mild conditions, the proposed 
modified MWQ algorithm converges to the optimal MWQ control despite the time-varying CSI and 
QSI. 

Index Terms 

Max Weighted Queue, Convergence Analysis, Queue Stability, Forster-Lyapunov, Stochastic 
Stability 
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I. Introduction 

Recently, there has been intense research interest studying cross-layer resource allocation of wireless 
adhoc networks for delay-sensitive applications. While the CSI indicates the transmission opportunity, 
the queue-state-information indicates the urgency of the packets in the queues. A good control policy 
(in delay sense) should strike a balance between the opportunity (CSI) and the urgency (QSI) and 
the design is highly non-trivial (T|-||6j. One approach, namely the Lyapunov Optimization technique 
(5J, j5J, allows a potentially simple control policy which adapts to the CSI and QSI. Specifically, the 
authors in J5J, (6} have proven that a max weighted queue (MWQ) throughput optimization solution 
can maximize the negative Lyapunov drift in the queue dynamics and it can achieve queue stability 
with reasonable delay performance. 

In most of the existing literature, it has been commonly assumed that the MWQ policy can be 
solved efficiently at each time slot based on the current realizations of CSI and QSI. However, this 
assumption may not be practical for moderate to large scale networks. Specifically, the MWQ solution 
requires solving a queue-weighted optimization problem Q, (6J and there is no closed- form solution 
in most cases. As a result, iterative algorithms (such as primal dual iterations) have to be used to 
obtain the MWQ solution at each time slot. While there is a lot of standard literature establishing 
the convergence of the iterative optimization algorithms, these works have assumed that the CSI 
and the QSI remains unchanged during the algorithm iteration^] However, for large scale networks, 
the algorithm iteration may involve not only the node itself but also over-the-air signaling between 
nodes. In this case, the CSI and the QSI may have changed after a few iterations and the existing 
convergence results (for static problems) failed to apply in this case of time-varying CSI and QSI. 
Furthermore, when the nodes in the adhoc network have limited power and computational resources, 
it may not be cost-effective for the node to iterate many times locally at each time slot as well. These 
observations motivate us to study the design and delay analysis of MWQ solutions in time varying 
wireless adhoc networks. 

In this paper, we consider a time-varying wireless adhoc network with power control driven by the 
MWQ algorithm. We study how the average delay performance of the MWQ solution is affected by 
the time-varying CSI and QSI. Unlike conventional works, we focus on the case where the MWQ 
algorithm iteration evolves in a similar timescale as the CSI and QSI dynamics. There are various 
first order technical challenges that have to be addressed. 

• Nonlinear Stochastic Algorithm Dynamics: One approach is to adopt continuous time con- 

1 Using the MWQ solution, the system of queues can be stable if the arrival rates are within the stability region J5J of 
the systems. 

2 In other words, it is assumed that the algorithm iteration time scale is much smaller than the CSI / QSI time scale. 
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trol theory and model the algorithm dynamics using deterministic ODE |[7|-pT|. In (7j-|(9j, 
the authors have considered the convergence behavior of the Foschini-Miljanic power control 
algorithm under time varying channels using the linear ordinary differential equation (ODE) 



approach. The authors in [10], |11| studied the tracking performance of the linear least mean 
square (LMS) algorithms under time varying channels. However, all these works have assumed 
linear and deterministic algorithm dynamics and these approaches cannot be easily extended to 
our case where the MWQ algorithm iteration is nonlinear and stochastic. 
Coupled Queue Dynamics and Algorithm Dynamics: The evolution of QSI depends on the 
control actions of the MWQ algorithm in each time slot. On the other hand, the evolution of the 
MWQ algorithm also depends on the time-varying QSI because the MWQ solution is obtained 
by solving a queue-weighted optimization. As a result of this mutual coupling, the techniques 



in our previous works |12|, which considered algorithm tracking performance where control 
decisions were made only based on CSI, cannot be easily extended in this case|^] 
• Delay Analysis and Compensation with Algorithm Tracking Errors: It is also quite challeng- 
ing to analyze and compensate for the delay penalty due to the MWQ algorithm tracking errors 
on the power control actions. In Q, (6J, the authors have derived an average delay bound for 
MWQ algorithm with i.i.d. CSI based on the Lyapunov drift analysis. However, this technique 
cannot be easily extended to our case when there are tracking errors in the MWQ control actions 
and correlations in the CSI evolutions. 
In this paper, we adopt a continuous time approach to model the algorithm dynamics of the MWQ 
power control iterations. We consider Markovian source arrivals and CSI evolutions so that the 
combined CSI, QSI and algorithm dynamics can be modeled by a stochastic differential equation 
(SDE). We show that studying the convergence behavior in the algorithm domain is equivalent to 
studying the stability property of a virtual stochastic dynamic system (VSDS). Using non-linear 
control theory and stochastic Foster-Lyapunov techniques, we establish a bound on delay performance 
due to time varying CSI and random source arrivals. Based on the VSDS dynamics, we propose a 
modification to the standard MWQ algorithm to compensate for the penalty due to the time- variation in 
the wireless adhoc networks. This paper provides a theoretical framework for studying the convergence 
of iterative algorithms as well as potential compensation techniques. The convergence analysis of 
iterative algorithms have widespread applications in network optimizations |3j, |5J, J6[ and signal 



processing 1 10 1, [11 1. 



3 In the previous work [ 12J, the control actions were made only based on the CSI. However, in this work, we consider CSI 
and QSI adaptive control policies and focus on the impact of both the time-varying CSI and QSI on the convergence of the 
algorithm. Here, there is a coupled dependency between the control actions (which depends on QSI) and the time-varying 
QSI (which depends on the control actions). This coupled dependency makes the problem challenging. 



Figure 1. Network topology. We consider a wireless adhoc network with N nodes and L links. We illustrate here N — 5 
and L — 6 as an example. The Z-th link transmits the l-th data flow. Transmission flows towards a same destination share 
the same frequency band and MUD and SIC are implemented at each receiving node to handle the inter-flow interference. 



Notations: A T (a T ) denotes the transpose of matrix (vector) A (a) and A H denotes the complex 
conjugate transpose. \x\ denotes the absolute value of x and ||x|| = maxjjxj} denotes the norm 
of vector x. For a complex variable z, Re [z] denotes its real part and z denotes its complex conjugate. 

II. System Model and Virtual Stochastic Dynamic Systems 

In this section, we shall first introduce the system model of the wireless adhoc network as well as 
the MWQ algorithm. Next, we shall introduce the notion of virtual stochastic dynamic system and 
establish the equivalence between the convergence behavior of the gradient algorithm and the virtual 
stochastic dynamic system. 

A. Network Topology, CSI and QSI Models 

We consider a wireless adhoc network with N nodes and L links, where each link corresponds to 
one transmitting and receiving pair, as illustrated in Fig. [T] Different receiving nodes occupy different 
frequency bands, while transmission flows towards a same destination share the same frequency band. 
Multiuser detection (MUD) and successive interference cancellation (SIC) are implemented at each 
receiving node to handle the inter-flow interference. The maximum achievable transmission rate at 
receiving node n is a set of rates \i[ that satisfy the following conditions p3| , 

£ m < log 1 1 + Yl N 2 w V5 ( n ) c £ ^( n ) (i) 

leS(n) \ leS(n) ) 

where C rev {n) is a collection of links whose destinations are at node n, S{n) are any non-empty 
subsets of C rev (n), hi is the channel fading coefficients of link I and pi is the normalized power 
allocated at link /. For example, Fig. [T] illustrates an example adhoc network with N = 5 nodes and 
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L = 6 links. £ rev (3) = {1, 2, 5} , £ re „(4) = {3,4,6}. Subscript I represents the link index as well 
as the flow index. 

We have the following assumptions regarding the channel state (CSI) hi(i). 
Assumption 1 (Temporally Correlated CSI Model): The support of the channel state process hi(t) 
is assumed to be h\ € % = {h G C : \h\ > ho} for some positive h$. Furthermore, hi(t) is a 
stochastic process described by the following reflective stochastic differential equation (SDE) in fl4[ 

1 

— ( 
2 



dhi = —-aihidt + afdwi + dvi, hi G % (2) 



where a\ determines the temporal correlation of hi(t), wi(t) is the standard complex Wiener process 



with unit variance, and dvi is the Skorohod reflection [14| term that satisfies dvi(t) > and 
f °° l{\hi(t)\ > ho}dvi(t) = 0. The fading process is independent w.r.t. the link index Z. ■ 
Note that hi(t) in ^ is a continuous time version of the auto-regressive (AR) process which has 



been widely used to model the dynamics of a correlated wireless fading channel 1 15 1. It captures the 
CSI variation speed that affects the convergence behavior of the algorithm in a time-varying channel. 
It can be shown that the process hi(t) has a stationary distribution. 

Incoming data packets randomly arrive at different nodes and are queued according to their 
destinations associated with particular transmission links. Let qi(t) and Ni(t) be the current queue 
backlog of queue and the number of packets arrived, respectively, at the Z-th queue at time t. We 
have the following assumptions regarding the bursty arrival process Ni(t). 

Assumption 2 (Bursty Source Model): The packet arrival Ni(t) is a Poisson process with intensity 
Aj. Specifically, Ni(t) follows a probability law given by, 

Pr (JV,(t + dt) - JV,(t) = l\N t (t)) = X t dt. 

■ 

The queueing dynamics of the wireless adhoc network can be described by the following SDE, 

dqi = -mdt + dNi. (3) 
The first term in ([3]) corresponds to the packet departure and the second term corresponds to the 



random packet arrival. Using Little's Law [16|, the average delay of the l-th link (Z-th flow) is given 
by Ti = q~[/\i, where q~i is the average backlog for the Z-th queue. As a result, there is no loss of 
generality to study the average queue length q t as this is proportional to the average delay. Obviously, 
the average queue length (or average delay) of the wireless adhoc network depends on how we allocate 
the transmit power pi (t) and data rate pi (t) of each link in the network. In the next section, we shall 
briefly review the MWQ algorithm, which is known to be a throughput optimal control (in queue 
stability sense). 
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B. Queue Stability and Max-Weighted Queue (MWQ) Algorithm 

There are different ways to control the power pi(t) and rates m(t) of the wireless networks but 
a reasonable algorithm (in delay sense) should adapt to both the CSI (to capture good transmission 
opportunity) and the QSI (to capture the urgency). In particular, we are interested in control policy 
that achieves a maximum queue stability region. We now first define the notion of queue stability, 
stability region and throughput optimal control. 

Definition 1 (Queueing Stability): A queue is called stable if limsup^^ j J * E [||q(r)||] < oo. 

■ 

The stability region C is defined as the closure of the set of all the arrival rate vectors {A/} that 
can be stabilized under some control algorithm that conforms to the power constraint E [p] G V 
PJ. A control policy that is throughput optimal is characterized in the sense that it stabilizes all 
the arrival rate vectors {A;} within the stability region C [17]. The throughput optimal policy is 
not unique and there are various known methods to achieve the maximum queue stability region. 
For technical reasons, we define a convex compact domain V = {p : < p < 2 LAmax //iQ}. Using 
Lyapunov techniques, a throughput optimal (in stability sense) formulation for the power and rate 
control actions at each time slot t is given in the following. 

Problem 1 (MWQ Formulation): 

max ZiMt)^i(t)-V Pl (t)} (4) 

subject to n(t) = {jn(t),...,fii(t)) T eC(p(t),h(t)) (5) 

where the physical layer capacity region C(p(t),h(t)) is a polyhedron defined by the constraints 
in ([TJ for all receiving nodes n. The parameter V acts as a Lagrange multiplier which controls the 
tradeoff between the average delay and the average power of the wireless network. ■ 

Note that the MWQ optimization problem in (|4j)-<[5]> is parameterized by the current CSI h(t) = 
{hi(t), . . ., h L (t)} and the QSI q(t) = {q\{t) . . . <7l(*)}- As aresult, the optimal solution p*(h(t), q(t)) 
and /i*(h(t),q(t)) of the MWQ problem is also parameterized by the CSI and QSI (h(t),q(t)). 

Due to the interference coupling in the MWQ problem, there are no closed form solutions for 
p*(h, q) and /x*(h, q) despite the problem in being convex. To solve the MWQ problem in 

(|4])-(|5]), we first have the following lemma regarding the rate allocation /i(p; h, q) given the power. 

Lemma 1 (Optimal rate allocation p8\f): Let 7r = {n(l), n(2), . . . vr(L)} be a permutation of the 
flow indices sorted in descendent order of the QSI qi, i.e. q w n) > q n (2) > • • • > QWz)- Given a power 
allocation p = [pi,P2> • • • ,Pl] , the optimal rate allocation solution of the MWQ problem in 
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is given by 



£rr(i) = log fl + |/^-(i)| 2 p^(i)) (6) 
/**(*) = log f 1 + 531^(1) | 2 P»r(0 J _1 °g f 1 + 53lM»)| 2 ^)V k = 2,...,L (7) 



Intuitively, given a power allocation, the optimal rate allocation vector (1 = . . . , is given 
by one of the vertices of the polyhedron C(p, h). In addition, the vertices are achieved by the SIC with 
decoding order n. As a result, finding the optimal /i(p; h, q) is equivalent to a linear programming 
problem, which requires L steps of iterations. Hence, we can focus on the power optimization in the 
MWQ problem given by 

L L 
max C(p(t);h(t),q{t)) = Y j q l (t)f H {j>(t);h(t),c l (t))-Vy j p l {t). (8) 

Using an iterative projected gradient search algorithm to find the optimal solution in ([8]), we derive 



the following power control algorithms dynamics [19|, 



p = K[V£(p;h(i),q(i))]£ (9) 

where k is a step size parameter, and the entry-wide projection operator is defined as [x]^ := 0, 
if z G dV is on the boundary of V and z + xdt € V, and [x]^ := x, otherwise. Hence, the queue 
dynamics of the wireless adhoc network under MWQ control is determined by the following coupled 
SDEs. 



dpi 



^ £(P;h ' q JI , 

1 



(10) 



dhi = --aihidt + afdwi + dvi (11) 
dqi = -A/(p;h,q)dt + diV;, V/ = 1,...,L. (12) 

In existing works, the convergence of the gradient algorithm in ([9]) and the throughput optimality 
of the MWQ in ^ are all based on an important assumption that the CSI and the QSI (h, q) remains 
constant during the algorithm iterations in Q. However, in practice, this may not be satisfied especially 
for fast fading channels and heavy traffic arrivals. In this paper, we are interested in studying the 
convergence behavior as well as the throughput and delay penalty of the iterative MWQ algorithm 
when the CSI and the QSI are changing at a similar timescale as that of the MWQ iterations. 



C. Virtual Stochastic Dynamic Systems 

In this subsection, we show that studying the convergence behavior of MWQ algorithm iterations 
in and the queue stability can be transformed into an equivalent problem of stochastic stability in a 



s 



Moving equilibrium 




Figure 2. An illustration of the algorithm trajectory for solving an MWQ problem with time-varying CSI and QSI. The 
dynamics of the CSI and QSI excite the equilibrium p*(t) to move around, and hence the convergence of p(t) is not 
guaranteed. 



virtual stochastic dynamic system (VSDS). As a result of this association, we can focus on analyzing 
the behavior of the VSDS instead of the original complicated MWQ algorithm dynamics. We first 
have a few definitions. 

Definition 2 (Equilibrium Point): Given the CSI and QSI parameter (h,q), p*(h, q) is called an 
equilibrium point of the MWQ algorithm dynamics in ^ if V£ (p*; h, q) = 0. ■ 
When the CSI and QSI (h, q) are quasi-static, the equilibrium point p*(h, q) is fixed and it has 



been shown 1 19 1, |20| that the MWQ algorithm iterations in (10) converges to p*(h, q) after sufficient 
iterations. However, when (h, q) are time- varying, the equilibrium point p*(h, q) is also time-varying 
as illustrated in Fig. [2] and it is not known if the MWQ iterations can track the moving target. To 
measure the tracking performance, we define the tracking error vector between the MWQ algorithm 
trajectory and the moving equilibrium point as below. 

Definition 3 (Tracking Error Vector): The tracking error vector of the MWQ algorithm is a vector 
difference between the algorithm trajectory p(t) and the target equilibrium point p*(h(i), q(i)), i.e., 

Pe(t)=p(*)-P*(t). ■ 

For a notation convenience, let tp : (h, q) i-> p*(h, q) be a mapping from the current CSI and 
QSI (h, q) to the equilibrium point p*(h, q). From Definition [3j the drift of the tracking error can 
be expressed as 

dp e = dp — dp* (13) 
= k [V£ (p; h, q)]p dt - ip q (h, q)dq - ^(h, q)dh 

where = q) an d iph(-) = gK^(h, q) are partial derivatives of the equilibrium point 

p* = ^(h, q) over the current QSI q and CSI h. They represent the sensitivity of p*(h, q) with 
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respect to the variations of the CSI and QSI (h, q). The terms ip q (h, q)dq and ^(h, q)dh represent 
the change of the optimal power dp* corresponding to the time- varying QSI dq(t) and CSI dh(t), 
respectively. Note that, as h is complex, ^(h, q)dh is defined as iphidhi = §^dxi + §^dyi, for 
each complex component hi = x% + iyi- Taking p = p* + p e , we denote 



f(Pe) h, q) = k [V£ (p e + p*; h, q; 



f 



as a mapping of the gradient iterations. Using the system dynamics of h(i) and q(i) in ( 1 1 )-( 12 1, we 



construct a stochastic error dynamic system to describe the tracking error process p e (t) as follows. 

Definition 4 (Stochastic Error Dynamic System (SEDS)): The stochastic error dynamic system is 
characterized by the following SDE 

dpe = fe(Pe] h, q)dt + b e (p e ; h, q)dN(i) + c e (p e ; dW(t), dV(t)) (14) 

where / e (p e ;h,q) = /(p e ;h,q) - ip q (.)fi(.) + \ij) h {.)Ah., b e {.) = -ij) q {.), and 

c e (.) = —^{^{A^dW^t) + dV(t)). A = diagjai, . . . , a^} is a matrix of CSI correlation coefficient 

in Q. ■ 

It is known that when the CSI and QSI (h, q) are static, the MWQ algorithm trajectory always 
converges to the static equilibrium point p*(h, q). However, when the CSI and QSI are time-varying 
in a similar timescale as the MWQ algorithm iterations, the algorithm convergence is not obvious. To 
study the behavior of the algorithm dynamics induced by the time-varying CSI and QSI, we construct 
a Virtual Stochastic Dynamic System (VSDS), which combines the overall dynamics of the CSI and 
QSI in ([TT]> and ([12]) with the Stochastic Error Dynamic System (SEDS) in ([14]) as follows. 

Definition 5 (Virtual Stochastic Dynamic System (VSDS)): Let z = (p e ,h, q) be a joint system 
state. The virtual stochastic dynamic system is characterized by the following coupled SDE, 



Z : 



where 



B(z) 



F(z) = 

02LxL 
II 



dz = F(z)dt + B{z)dN + C(z, dW, dV) 
/(p e ; h, q) - ^ q fi{p e + p*) + \ip h Ah. 

-A(Pe + P*) 



(15) 



and C(z,dW(t)) 



-ip h (h,q)(A'dW + dV) 
AUw + dV 

Ol 



4 The complex derivative for a real value function ip(h) is defined as 
for h = x + iy. The Taylor expansion of ip(h) is thus given by dip 



dh 



1 (d± _ -d±\ 
1\Bx dy ) 



<>y 

ox oy 



dy (2l). 
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Figure 3. An illustration of the connection between the MWQ algorithm dynamics and Virtual Stochastic Dynamic System 
(VSDS). Fig. (a) illustrates the dynamics in the MWQ algorithm domain. The control policies (/^(t), p(t)) from the MWQ 
algorithm iterations are driven by the CSI dynamics h(t) and the QSI dynamics q(t). Fig. (b) illustrates the coupled MWQ 
iterations, CSI and QSI from the VSDS perspective, where the power tracking error p e (t), the CSI and QSI (h(t),q(t)) 
are modeled as a joint state of the SDE, which is driven by external stochastic processes Wt and Nt- 



Fig. [3] illustrates the inter-connection between the key components in the VSDS. Fig.|3ja) illustrates 
the dynamics in the MWQ algorithm domain. Specifically the queueing dynamics q(t) is driven by 
the bursty arrival process N(t) as well as the control policy p(i)). At the same time, the 

control actions (/z(i),p(i)) are driven by the MWQ algorithm iterations, which depend on the CSI 
h(t) and the QSI q(i). Fig. [5Jb) illustrates the dynamics in the VSDS domain. The system consists 
of the SEDS (driving the tracking error process p e (t) = p(t) — p*(t)) as well as the CSI h(t) and 
QSI q(t) driven by external processes W(i) and N(i). 

We show in the following theorem that, studying the convergence behavior of the MWQ algorithm 



(j9j) is the same as studying the stability property of the VSDS in ( |T5| ). Also, evaluating the stability 
of queue backlogs driven by the MWQ algorithm dynamics is equivalent to investigating the stability 
property of the system state z(i) in the VSDS. 

Theorem 1 (Connections between the MWQ Algorithm Dynamics and the VSDS): The actual queue 



trajectory of the MWQ algorithm in ( 12 1 is the same as the solution process q(i) in the VSDS in ( 15 I. 
Furthermore, the power control trajectory of the MWQ algorithm in (TTOb converges to the equilibrium 



p*(h, q) if and only if the SDE in (14i is globally asymptotically stable at p e = 0, i.e., given any 
initial state p e (0) G M+, lim^oo Pr (p e (£) = 0) = 1. ■ 
Proof: Please refer to Appendix [A] for the proof. ■ 
As a result of Theorem [T] we can focus on the VSDS dynamics in order to study the delay 
performance penalty of MWQ due to time varying CSI and QSI. Nevertheless, due to the mutual 
coupling of the SDEs in the VSDS, it is still difficult to study its stability behavior. In the rest of the 
paper, we will focus on extending the stochastic Foster-Lyapunov method 1 22 ] to derive the stochastic 
stability results of the VSDS. 
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III. Performance Analysis of MWQ Algorithm under Time-Varying Arrivals and 

Channels 

In this section, we shall analyze the tracking performance of the MWQ algorithm under time- 
varying channels. We bridge the connection between the property of the Lyapunov stochastic drift 
and the stochastic stability of the corresponding VSDS. Following this result, we then derive an 
expected queue bound under the MWQ algorithm in time-varying channels. 



A. Stochastic Stability of Random Process 



Let z = (p e ,h, q) be a joint state of the VSDS in (15), where p e is the tracking error, h is the 
channel coefficient and q is the queue backlog. Denote z(i) as the stochastic process starting from 
t = with initial state z(0). We have the following definition of stochastic stability to characterize 
the behavior of z(i). 

Definition 6 (Stochastic Stability): Given any initial state z(0) G Z, the stochastic process z(t) is 
globally stochastically stable, if there exists < D < oo, such that 



1 f* 

lim sup - / E||z(i)|| < D. 

t— >oo t Jq 



Notice that this definition is analogue to the usual concept of stability in deterministic system [23 1, 
whereas, the condition here is taken over a time averaged expectation. This general criteria can be 
applied to a non-stationary process, such as a queueing system with different classes of services. In 
fact, in this work, we do not require the queue dynamics and the MWQ algorithm trajectory to be 
stationary. 

Define a Lyapunov function of the state z(t) as V(z) = z^z. We can investigate the evolution of 
the Lyapunov function by studying its drift along the state trajectory. Analogue to the discrete-time 
one-step conditional Lyapunov drift in (5J, we define the continuous time Lyapunov drift generator 



as 



= lim nvw + s))-v Wm t)\ 

54.0 

where the expectation (conditioned on the current state z(i)) is taken over the randomness of the 
CSI and the arrival to the QSI. The Lyapunov drift represents the expected evolving direction of the 
Lyapunov function V(z) from the current state z(t), and LV(z) is called an infinitesimal estimator 
of V(z). As V(z) is a norm-like function [23], the boundedness of the Lyapunov function implies 
the boundedness of state z(t) and the dynamics of the Lyapunov function reveals the evolution of 
state z(t). For example, when the drift is negative, ||z(i)|| is most likely decreasing. The Lyapunov 
drift LV(z) can also be derived from the SDE of z(t) as stated in the following lemma. 
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Lemma 2 (Continuous-Time Lyapunov Drift ft24\/): Suppose that there is a d-dimensional stochas- 
tic process z(i) described by a SDE 

dz = f(z)dt + s(z)dW + h(z)dN 

where W(t) G C L is a standard complex Wiener process and N(i) = (iVi(i), . . . , Ni,(t)) G is 
a Poisson process with intensities Aj, Z = 1, . . . , L. For any given Lyapunov function V(z) € C 2 : 
Z — > M + that has compact support [24], the stochastic Lyapunov drift can be written as 



LV(.) = ^/M + K 



g(z) H —^g(z)+g(z)—^g(z) H 
ozoz ozoz 



+ (v(z + h«\z))-V(z)) 

l=i 



where h^ l \z) is the Z-th column of h(z). ■ 
The above lemma establishes a connection between the infinitesimal estimator LV(z) and the 
specific SDE. The proof is similar to that in |24| with a notation extension to complex variables 



\2\\. By exploiting the property of the Lyapunov drift, we can characterize the stochastic stability 



of random process z(t) described by the SDE in (15 1. We summarize the result in the following 
theorem. 

Theorem 2 (Stochastic Stability from Lyapunov Drift): Suppose the stochastic Lyapunov drift of 
the process z(t) satisfies 

LV(z)<-a\\z\\+g(s) (17) 
for all z G Z, where a is some positive constant and s(i) is a stochastic process that satisfies 

i r* 

lim sup - / E[g(s(T))]dT < d 

t^roo t JO 

for some function g : s t- > R and d < oo. Then the process z(t) is stochastically stable, and 



If* d 
lim sup - / E||z(r)||dr < -. 
t— >oo t Jq a 



Proof: Please refer to Appendix [B] for the proof. ■ 
The above result is an extension of the Foster-Lyapunov criteria for continuous time processes 



in \22\. If z(t) is a system state that relates to the queue length and the tracking error of the 
power variable, the stochastic stability forms an estimation on the queue bound as well as the power 
penalty due to the time-varying parameters. The advantage of the Foster-Lyapunov method enables 
a qualitative analysis of the VSDS without explicitly solving the SDE. In the following, we shall 
illustrate how to construct a Lyapunov drift for the VSDS. 
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B. Stability Analysis of the MWQ Algorithm 

In this section, we shall apply the stochastic stability analysis method to study the stability of the 
VSDS (15). Specifically, according to Lemma [2J the Lyapunov drift for the VSDS (15 1 is given 



1 L 
LV(z) = 2z H F(z) + -tr[2C(z,.) H C(z,.)] + J] A, [v(z + B^) - V(z) 

l=i 

< 2pf /(p e ; h, q) - 2p2V ff (h, q)A(p e + P*) + PeM^, q)Ah 



L 



-h H Ah - 2p^ q (h, q)A - 2 m MPe + Pi ~ A;] 



2=1 

L 



+11(2^^^^^+^^} + J2(2ai + \i) (18) 
^ ' 1=1 

where flW stands for the l-th column of B(z) and A = diag{Ai, . . . , Xl} is the arrival matrix. 

As illustrated in Theorem |2j negative drift terms in LV(z) are desirable because they can drive 
the stochastic state process towards the origin and contribute to stabilization of the VSDS. In the 



following, we shall analyze the key terms on the R.H.S. of (18 1 and discuss their contributions. 
Intuitively, a fast MWQ algorithm and a high transmission rate can contribute to driving the LV(z) 
negative and we shall elaborate on such properties in the following. 

We first define the following, which will be used throughout the analysis. Let S m i n (t) = min; 
and S max (t) = max; ||/i;(^)| 2 | be the minimum and maximum channel gains among L transmission 
links at time t, respectively. Notice that has stationary distributions and we denote its cumulative 
distribution function as F^{x). Thus 5 m i n and 5 max are also ergodic processes with stationary 
distributions given by the L-th order statistics as Fg im (s) = P{S m - m < s) = 1- [l - P (|/i| 2 < s)] L = 
1 - [1 - F h (s)} L and = P (S max < s) = P (\h\ 2 < s) L = F h (s) L . 

The following lemma summarizes the contribution of the convergence speed of the MWQ iterations 
in ^ to the drift LV(z) in 



Lemma 3 (Negative Drift Contribution of Convergence Speed in MWQ Gradient Iteration): Given 
any CSI and QSI realizations h(t) = h and q(f) = q >- 1, there exists a(S m - m , S max ) > that 
satisfied 

Pr/(p e ;h,q) < -Ka||p e || 2 (19) 

for all t > 0, where k is the step size parameter of the MWQ iterations in Q. ■ 
Proof: Please refer to Appendix [C] for the proof. ■ 

5 Note that Lemma [2] does not specify the drift for the term dV. However, as the reflection \dV\ < \A^dW\, we can 
treat V(i) as A^Wff) and yield an upper bound for LV(z). 

6 For vectors a = (ai, 0,2, . . . , a^) and b = (b\, 62, • • • , a >- b is defined as a; > bi, Vi = 1, . . . , L. 
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This lemma illustrates that the tracking error term p e contributes to the negative drift (proportional 



to na) in LV(z) in ( 18 1. The larger the tracking error ||p e || is, the stronger force the MWQ iterations 
will drag the system state p to the optimal p*, which in turn helps the stabilization of the system. 
In fact, the negative drift depends on the MWQ iteration step size k, which controls the convergence 
rat J^of the MWQ iterations under static h and q. 



From the Lyapunov drift for VSDS in (18), the transmission rate fi(t) also contributes to negative 
drift in (18 1, which in turns help to stabilize the VSDS. Before we quantify the negative drift 
contribution, we first discuss several structural properties of the transmission rate at the equilibrium. 
Let /x* (h, q) = ft (p*(h, q); h, q) be the transmission rate at the equilibrium point p*(h, q) (optimal 
transmission rate under the MWQ policy). We have the following lemmas about the structural property 
of £t*(h, q) and the actual transmission rate 

Lemma 4 (Structural Properties of the Transmission Rate at Equilibrium): The transmission rate 
/x*(h, q) at the equilibrium p*(h, q) of the VSDS has the following properties, 



^%M?( h >q) > ||q||min|log ^?p||q||J , LA max + log j 



(20) 



and 

iminjlog f-^||q||^ , LA max + log | < ||/Lt* (h,q) || < log ^^p^||q||^ (21) 

for ||q||5 m i n > V. ■ 

Proof: Please refer to Appendix [D] for the proof. ■ 
Lemma 5 (Structural Properties of the Actual Rate p,(t)): There exists a (3 > depending on 
5 m i n and S ma _ x , such that, for all t > 0, the actual transmission rate at time t, p,(t) = /i(p(i); h(t), q(t)) 
satisfies 

||/i*(h(t),q(t))l|-log(l + |9||p e (*)||)< \\m\\ < ||/**(h(t),q(*))ll+log(l + ^||p e (t)||). (22) 



Proof: Please refer to Appendix [E] for the proof. ■ 
Lemma [4] shows that the term Y^1ll JL * grows faster than ||q|| and ||/i*(h, q)|| is lower bounded 
with the order log(||q||). On the other hand, Lemma [5] illustrates that, although there is a tracking 
error p e in the power allocation, a minimum transmission rate is still guaranteed and the rate penalty 
due to the tracking error p e is no larger than log (1 + /3||p e ||). 

7 Note that the MWQ iteration in J9J is expressed in continuous time and a larger k is always desirable from the perspective 
of convergence speed. However, in practice, the MWQ iterations are implemented in discrete time and the corresponding 
discrete time step size is given by kt where r is the slot duration of iterations. For a given r, a large « will speed up the 
iteration but also contributes to a larger steady state errors of O(kt) in the discrete time iterations. 
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Based on the properties in Lemma Hpl we can derive an upper bound of the Lyapunov drift in 



(18). Let a a = \\A\\ = max {a;} be the CSI fading rate parameter, where A is the coefficient matrix 
of the CSI dynamics defined in §2^. Let A max = ||A|| = max; {A;} be the maximum arrival rate 
among all the transmission links. 

Lemma 6 (Lyapunov Drift Property for VSDS): Suppose there exists constants 7 g < oo and 7^ < 
00, such that ||^ g (h(t), q(t))[| < jq, \\iphO^-(t), q(*))ll — 7/i> f° r au t > O.ln addition, the step size k 



satisfies k > - max {7^, fi'jq} under all h(i). Then the stochastic Lyapunov drift in ( 18 1 is bounded 
by 



where 



LV < -(||p e || + ||q|| + ||h||) + Z)(5 min ,S max ) (23) 



^(Smin, Smax) = L(2a A (l + Jh) + lq + Vax) + ^~T, 2 



KCH 'S'min 

and g(S mm , S max ) is a function bounded for all S'min and S, 
Proof: Please refer to Appendix |F] for the proof. 



8oa[1 - %a A /(Ka)] 
2 7 2 A max V LX _, 

) + c 



From the above lemma, the Lyapunov drift (18 1 is increasingly negative for sufficiently large ||q| 



and ||h|| and this negative drift drives the system state back to a trajectory with bounded norm. 
This property stabilizes the VSDS. Denote ao = E [^], 7 = E g[i-7 2 a A /(Ko:)] > & = ^[^r~~ ] an( ^ 
g = E [g(S m i n , Smax)]. Note that as S'min and S max are the L-th order statistics of stationary processes 
\hi\ 2 , a and g are bounded above. The stability results of the VSDS can be summarized as follows. 



Theorem 3 (Stability of the VSDS): The system state z(t) of VSDS in (15) is stochastically stable 
and satisfies 

^ 2^2 

limsup^ f E[||z(r)||]dr<L(2a A (l+7l)+7 9 2 +Amax) + ^ + ^ L ^+^2 iA — ^a+g. (24) 

t^oo t Jq h CLA K 

■ 

The above theorem is a direct result of Lemma [6] and Theorem [2j As ||q|| < ||z||, we can obtain 
the average queue bound from the following corollary. 

Corollary 1 (Expected Average Queue Bound under Time-varying CSI and QSI ): The expected av- 
erage queue bound under MWQ algorithm in time-varying CSI and QSI is given by 

^ 2 \2 

limsup I [ E[||q(r)||]dr<L(2aA(l+ 7 ^)+7 g 2 +Amax) + ^ + ^ J ^+^2 LA — (25) 
t->oo t Jo a A K 



The result shows the upper bound of the average worst case queue (corresponding to the worst case 
delay) of the network. The bound depends on several important parameters, namely the CSI fading 
rate a a, and the sensitivities of the equilibrium p*(h, q) w.r.t. h and q, (7^, 7 9 ). 
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Figure 4. A numerical illustration of the average queue bound (for the worst queue) in ( |25fr versus the CSI fading rate a a- 
The numerical result is under L — 4 links and maximum arrival rate A max = 1, and different assumptions of sensitivity 
parameters jh and j q . The numerical queue bound is increasing with the CSI fading rate a a, the parameters 7^ and -y q . 



Fig. [4] gives a numerical illustration of the theoretical queue bound in (25 1 under L = 4 links, 



maximum arrival rate A max = 1, various CSI fading rates cla, and sensitivity parameters 7^ and j q . 
Note that the delay bound increases w.r.t. 7^ and 7 5 . Note that the delay bound increases w.r.t. 7^, 
7 g and at both large and small fading speed (cla). For large a a, there is the penalty of the increased 
tracking error due to time varying CSI. For small ai, the delay increases because the CSI may be 
stuck at a poor state for quite a long time. 

IV. Adaptive Compensation for the MWQ Algorithm in Time- varying Arrivals and 

Channels 

Based on the stochastic dynamics modeled by the VSDS, we consider modifying the gradient MWQ 
iterations in ([9]) to reduce the penalty induced by time varying CSI and QSI. Specifically, we introduce 
a compensation term to improve the stochastic dynamics of the VSDS in ( [T5] ). This corresponds to 
a compensation term in the MWQ algorithm to offset the effect from the time-varying CSI and QSI. 
The overall compensated MWQ algorithm is shown to have a better convergence robustness w.r.t. 
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time varying CSI and QSI both analytically and numerically. 



A. A Proposed Algorithm with Compensation Term 



We have shown in Section II-C that the dynamics of tracking error p e (t) can be modeled by a 



stochastic error dynamic system in ( 14 1, which consists of a drift term / e (.) and diffusion terms 6 e (.) 
and c e (.;dW). Without the diffusion terms, the SEDS eventually converges to the origin, as / e (.) 
contributes a negative drift to the infinitesimal estimator LV(p e ), where we define the Lyapunov 
function as V(p e ) = pjp e - However, with the presence of the diffusion terms, the system dp e = 
/ e (p e ;h, q)dt is disturbed from the equilibrium at p e = and the state p e is driven away from 
the origin. The magnitude of b e (.) and c e (.) reflect the chance and intensity that the state p e (i) is 
being disturbed. Based on this observation, one way to stabilize p e (i) is to offset the diffusion terms 



b e (.) and c e (.) in the SEDS dynamics in ( 14). Equivalently, this corresponds to modifying the MWQ 
algorithm iterations in d9l) to compensate for the effects of time varying CSI and QSI. From the error 



tracking vector dp e in ( 13 1, we would like to compensate the movement of the optimal target dp*(t) 



so that the resulting SEDS becomes dp e = n [V£ (p e + p*;h, q)]p dt. In this ideal case, the p e 
will converge to 0. However, the challenge is that we do not have an exact expression for dp*(t) 
during the iteration because we do not have closed form expression of the equilibrium p*(t). We 
shall propose an indirect method of estimating the compensation term. 

Since the MWQ problem in ([8]) is convex, p* is the optimum if and only if there exists A* >z 0, 
such that 



V£(p*;h,q) + A* 







Vl = l,...,L. 



(26) 
(27) 



We denote the above system of equations (KKT conditions) as ^(x*; h, q) = 0, where x* = (p*, A*) 



d<S> 



Note that x* is unique for a convex problem. Using implicit function theorem and assuming g—; 
non-singular, we have 



is 



dx* 



dp* 
d\* 



\dx* 



<9q 



dq - 2Re 



\dx.* 



-i 



d<5> 
dh 



dh 



(28) 



As a result, we obtain dp* = (p q {p* , A(p); h, q)dq + Re [ifh(p*, A(p); h, q)dh], where the vector- 
valued functions ip q (p*;.) and <fh(p*',') are the rows for primal variable dp* from — (J^r) 1 ' 1 



9q 



and -2 



d<S> 
cbc* 



d<S> 



^ in (28 1, respectively. Thus the MWQ iterations with compensation is given by 



p = [kVC( P ; h(t), q(i)) - q (p, A(p); h(t), q(t))dq - Re [^(p, A(p); h(i), q(*))dh]£ (29) 



where ip q (.)dq and Re[^/j(.)dh] are compensation terms. Here, we use the current algorithm state 
p(t) as an approximation of the target equilibrium p*(t) and A is computed via the KKT conditions 
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in (26l-(27). The compensation term can be interpreted as an estimation on how the target equilibrium 
p* is moving according to the time-varying CSI and QSI (dh,dq). When p is close to p* (i.e., p e 
is small), the estimation <p(p, A(p); dh, dq) on dp* is accurate. Thus the compensation term helps 
further reduce the tracking error and the algorithm would eventually converge to the equilibrium 
p*. We shall investigate the convergence behavior of the compensation algorithm in the following 
subsection. 

B. Performance Analysis for the Compensation Algorithm 

Suppose the functions (p q (p;.) and c/?/j(p;.) are Lipschitz continuous, i.e., there exists positive 
constants L q , L h < oo, such that \\<p q (p; .) - q (p*; .)|| < -Mp-P*|| and \\0 h (p; .) - tp h (p* ; ,)\\ < 
L^||p — p*||, for all p € M+. Let ^ max be the maximum transmission rate that the system can support 
and a > be defined in ( fT9] ) uniformly for all CSI realization h. The following theorem provides a 
sufficient condition to the convergence of the compensation algorithm. 

Theorem 4 (Convergence of the Compensation Algorithm): Provided that the step size parameter 
k satisfies, 



1 

K > - 

a 



(/^max + ^max^) L q + l^^q + ^ a Al^h 



for all t > 0. Then the MWQ iterations with compensation in (29) asymptotically tracks the moving 
equilibrium point p*(t) with no errors, i.e., Ve > 0, 

limPr[||p(i)-p*(i)|| <e] = l. 

■ 

Proof: Please refer to Appendix [G] for the proof. ■ 
Theorem |4] shows that when a large enough step size k is available, the compensation algorithm can 
converge to the equilibrium point p*(t), and there is no performance penalty due to the time-varying 
CSI and QSI. The convergence is affected by the parameters L, a a, and L q , where L is number 
of transmission links in the network (the system dimension), a a is the CSI variation rate of the whole 
network, and Lh and L q represent the sensitivity of the equilibrium point p* (t) w.r.t. the time-varying 
CSI and QSI. On the other hand, for conventional gradient iteration in Q, the algorithm cannot have 
p e — > no matter how large the iteration step size k is used. This is due to the fact that the target 
equilibrium p*(t) is moving due to the time-varying CSI and QSI. 

Remark 1 (Interpretation of the results): In practice, we would like to implement the modified 



MWQ iteration in (29) on discrete time. The iterations of (29 1 can be written as 



p(t + r) = {p(t) + KTV£(p(t); h(t), q(t)) - q (.) Aq(t) - Re [0 h (.)Ah(i)]}J 
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in discrete time where Ax(t) = x(t + r) — x(t) and r is the time step. In this case, the overall error 
between p(t) and p*(t) is contributed by (a) algorithm convergence error and (b) steady state error. 
While Theorem [4] suggests that a large step size k is always desirable from the algorithm convergence 
error perspective, the above analysis did not consider the steady state error (due to constant step size) 
o(kt) associated with discrete-time implementation. The overall impacts of steady state errors and 
tracking errors will be demonstrated in the numerical results section. 

V. Numerical Results and Discussions 

In this section, we shall simulate the tracking performance of the conventional MWQ iteration and 
the proposed compensated MWQ iteration in time-varying channels. We also demonstrate the delay 
performance for the two MWQ iterations under various CSI fading rates. We consider a wireless 
ad-hoc network with 5 nodes and 6 links as depicted in Fig. [T] The Z-th link transmits the Z-th 
data flow. Transmission flows towards a same destination share the same frequency band and SIC 
is implemented at each receiving node to handle the inter-flow interference. The CSI hi for each 
link is modeled by a unit variance Markov process described by the SDE in (|2]). Data arrivals are 
modeled by continuous time Poisson processes with rate A = 20 packets/second. All the algorithms 
are implemented in discrete-time iterations with simulation time step 1 ms and the queueing system 
was run over a time duration T = 100 min. The delay performance of the conventional MWQ 
iterations in Q and the modified MWQ iterations with compensation in (29 1 are compared against 
the following reference baselines. 

• Baseline 1 - Constant Power Allocation: At each time slot, fixed power P is allocated to each 
link and the transmission rate is computed by Q-Q. 

• Baseline 2 - Throughput Optimal Power Allocation: The throughput optimal power control 
is computed by solving the MWQ problem in to obtain the target equilibrium p*(h, q) 
at each time slot t. 



A. Power Tracking Performance of the MWQ Iterations 

Fig. [5] captures the power control algorithm trajectory p(i) versus time at a CSI fading rate of 
cla = 200. The algorithms update on every r = 1 ms time slot and the step size is chosen to be 
0.5 (corresponding to k = 500 sec -1 for continuous-time trajectory). Throughout the simulation, 
the average delay is measured as T\ « 500 ms. As illustrated, the target equilibrium p* (t) changes 
significantly over time due to the time varying CSI. The conventional MWQ iterations PMWQ,i(t) 
fail to track the moving target p*(t) accurately but the trajectory of the compensated MWQ iterations 
Pcom,i(t) can track the moving target quite well. 
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Figure 5. The power control algorithm trajectory p(t) versus time at a CSI fading rate a,A = 200 and packet arrival 
rate A = 20 packets/second. The algorithms update on every r = 1 ms time slot with step size 0.5 (corresponding to 
ft = 500 sec -1 ). The average delay is measured to be Ti ss 500 ms. As illustrated, the target equilibrium p*(t) changes 
significantly over time due to the time varying CSI. The conventional MWQ iterations PMWQ,i(t) fail to track the moving 
target p*(i) accurately but the trajectory of the compensated MWQ iterations p CO m,i(t) can track the moving target quite 
well. 

Fig. [6] illustrates the average tracking error of the power trajectory p(t) versus the fading rate cla- 
The average tracking error of the power trajectory is defined as e = ^ J Q T \\p(t) — p*(t)\\dt. It is 
shown that the average tracking error of conventional MWQ iterations increases with the fading rate 
a a- On the other hand, the tracking error of the modified MWQ iterations (with compensations) is 
much smallei^] than that of the conventional MWQ iterations. 

B. Power-Delay Tradeoff Performance 

Fig. [7] illustrates the per-node average power versus the average delay at different fading rates. 
Note that along each curve, we have different values of V, which acts as a tradeoff parameter for 
power-delay tradeoff. Small V corresponds to small delay and vice versa. Observed that to maintain 
the same average delay of 2 seconds, the conventional MWQ iterations require 2.3 dB more power 

8 Note that the tracking error shown is the overall error obtained using discrete-time iterations, which include the errors due 
to algorithm convergence and steady state errors (due to constant discrete time step size). From Theorem [4] the algorithm 
convergence error tends to zero for the modified MWQ but there is a steady state error in Fig. [6] due to the constant step 
size in discrete time implementation. 
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Figure 6. The average tracking error of the power trajectory p(t) versus the fading rate a a under packet arrival rate A = 20 
packets/second. The algorithms update on every r = 1 ms time slot with step size 0.5 (corresponding to k = 500 sec" 1 ). 
The average tracking error of conventional MWQ iterations increases with the fading rate a a- On the other hand, the 
tracking error of the modified MWQ iterations (with compensations) is much smaller than that of the conventional MWQ 
iterations. Note that the error consists of contributions from both the algorithm convergence error and steady state error due 
to constant step size (in discrete time). From Theorem [4] the algorithm convergence error of the modified MWQ converges 
to zero but there is still residual steady state error. 



than the throughput optimal scheme. On the other hand, the proposed modified MWQ algorithm 
with compensation suffers from a very small power penalty (< ldB) compared with baseline 2 (the 
throughput optimal scheme). Furthermore, as the CSI fading rate aA increases, the conventional MWQ 
iterations eventually require as much power as baseline 1 (constant power allocation) does, while the 
proposed modified MWQ algorithm with compensation still has a reasonable power gain compared 
to baseline 1. 

VI. Conclusions 

In this paper, we have analyzed the convergence behavior and the queue delay performance of the 
conventional MWQ iterations in a wireless adhoc network, in which the CSI and the QSI are changing 
in a similar timescale as the algorithm iterations. We first show that the algorithm convergence can 
be captured by studying the stochastic stability of an equivalent virtual stochastic dynamic system 
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Figure 7. The per-node average power versus the average delay at different fading rate. Observed that to maintain the 
same average delay of 2 seconds, the conventional MWQ iterations require 2.3 dB more power than the throughput optimal 
scheme. On the other hand, the proposed modified MWQ algorithm with compensation suffers from a very small power 
penalty (< ldB) compared with baseline 2 (the throughput optimal scheme). Furthermore, as the CSI fading rate oa increases, 
the conventional MWQ iterations eventually require as much power as baseline 1 (constant power allocation) does, while 
the proposed modified MWQ algorithm with compensation still has a reasonable power gain compared to baseline 1. 

(VSDS). By extending the Foster-Lyapunov criteria, we established the technical conditions for queue 
stability and derived the associated queue bounds. Based on these analyses, we have proposed a novel 
adaptive MWQ algorithm with a predictive compensation to counteract the effects of the time varying 
CSI and QSI. We have demonstrated that with some mild conditions, the modified MWQ iterations 
(with compensation) can converge to the moving target power p*(i) despite the time varying CSI 
and QSI. Finally, simulation results demonstrated the performance gain of the proposed algorithm in 
both the network delay performance and the tracking error of the power trajectory. 



In this section, we give a brief introduction to the Lyapunov method for algorithm convergence 
analysis, which motivates us to connect the algorithm trajectory to the VSDS. 



Appendix A 



Connections between the Optimization Algorithms and the VSDS 
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We focus on gradient-based methods that are widely used for computing the optimal resource 
allocations in wireless communication networks and are well-suited for implementations across a 
distributed network. The gradient method searches the optimum point x* of the objective function 
£{x) following 

dx 

x = — = K 



dC 



dx 



Here we study the convergence behavior by constructing the tracking error dynamics of the algorithm 
trajectory. Define the tracking error x e = x — x* and substitute it into the above dynamics, we obtain 



dC(x e + x*) 

dx e . 



T 



(30) 



Hence the convergence analysis is transferred to stability analysis (23j of the virtual error dynamic 



system (30 1 at the origin x e = 0. 



A classic method to study the stability of a dynamic system is via the Lyapunov theory [23]. We 
first construct a Lyapunov function which has the following properties, 

V(x e ) — > oo, as ||x e || — > oo, and V(x e ) — > 0, as ||x e || — > 0. 



The Lyapunov theory says, if V(x e ) < for all x e S M n \{0}, then the dynamic system x e = f(x e ) 
is asymptotically stable at the origin x e = |23| . 

Note that, the objective function C(x;h(t),q(t)) we focus on in this paper has stochastic time- 
varying parameters h(t) and q(t), which may evolve in a similar timescale to the algorithm trajectory. 
We tackle this problem by constructing the VSDS from the algorithm dynamics, and extending the 
Foster-Lyapunov criteria (in Theorem [2]). We show the connection between the algorithm trajectory 
and the VSDS in the following. 

Proof of Theorem^ Note that the VSDS in (15) consists of three components, p e , h and q, where 



the dynamics of h(i) and q(t) are just the same as (11) and (12). We only need to show that the 
dynamics of p e (i) in the VSDS in ( 15 1 implies the MWQ power control algorithm dynamics of p(t) 
in ( [TO) ). Equivalently, we need to show 

p(i) = p e (0)+ / d Pe (r)+p*(t) 
Jo 

to be the solution of ( fT0| ). On the other hand, by the definition of tracking error (Definition [3]), 
p e (0) + / * dp e (r) + p*(i) = p e (t) + p*(i) = p(i). Therefore, by substituting p e (t) + p*(i) with 



p(i) in the VSDS in (15), we see that the trajectory q(i) in the VSDS is just the same as that in 

To prove the second part of the theorem, we consider that there is no disturbance applied to the 
SEDS in ( p"4| ) by considering dN = and dW = 0. Equivalently, we take dh = and <iq = in 



( fT3| ). The SDE of p e (i) in ([T_4]) reduces to 

dp e 



k[VC (p e + p*;h,q)]+ +p .dt. 



(31) 
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By the definition of equilibrium point (Definition [2]), p e — > corresponds to V£ — > 0. Hence the 
origin is an equilibrium to the SDE in p4] ). On the other hand, if the origin is an equilibrium to 



(31 1, p* = ip(h, q) must be the equilibrium to the dynamics of the power control algorithm in ( 10 1. 
Hence, we complete the proof. 



Appendix B 
Proof of Theorem [2] 

Proof: Define a sequence of stopping time t n = inf {t > : z(i) > n}. By Dynkin's formula 



0<V(z(t n )) <V(z(0))+E 



(-o||z(r)||+^(s(r)))dr 



Hence we have 



E 



a\\z(T)\\dT 



< V(z(0))+E 



g(s(r))dr 



Exchanging the order of integration and expectation, we have 



i E\\z(r)\\dT < + ±- f tn E[g(s(r))]dr 

In JO <> n a l n Jq 

Taking limit on both sides, we obtain 

lim sup - [ U E\\z(r)\\dT < lim sup (-YM^L + 1 f " Ie[ 5 (s(t))A < -. 

n— >oo t n Jq n— >oo \t n CI t n J q CL J CI 

Notice that t n — > oo as n — > oo, and V(z(0)) is bounded. Thus the result holds. 



Appendix C 
Proof of Lemma[3] 

Proof: According to Lemma [TJ the optimization problem ([8]) can be written as 

max 97r(1) log (1 + | 2 p^ ( i)) (32) 

+<?tt(2) log (l + 1^(1) | 2 ^(1) + |^vr(2) | 2 Ptt(2)) ~ log (l + 1^(1) l^l))] + ••• 



log ^1 + \K(i)\ 2 Pnii^j ~ log f 1 + ^ \K 



(i) 



for a certain permutation 7r, where ^(fc-i) > q^ik)t for A; = 2, . . . , L. As the objective function 
£(p;h, q) is a combination of logarithmic functions, it can be verified that £(p;h, q) is strictly 
concave in p and V 2 £(p; h, q) < 0. In addition, as the domain V is compact and under the condition 
that q^u) > 1, there exists a positive constant a(5 m i n ,5 max ) > depending only on the channel 
gain parameters 5m in (t) and 5 , max (t), such that the Hessian of £(p; h, q) satisfies V 2 £ ^ — al for 
all p G V. 
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Based on this observation, we obtain, 



p^/( Pe ;h,q) = p^/(0 + ;h,q)+p^ / V/(Cp e ;h,qKPe (33) 

Jo 

= p^/(0 + ;h,q) + pJ / KV 2 £(£p e + p*(h,q);h,q)d£p e (34) 

J o 

< - f aK||p e || 2 ^ (35) 



-aK||p e || 2 



where p^/(0 + ;h, q) = (p — p*) T V£(p*(h, q); h, q) < is the optimality condition for p*(h, q) 
in the optimization problem ([8]). The equality (|33])is from Taylor expansion of the gradient iteration 



function /(.), the second equality (34i is from the fact that V/ = V 2 £, since / = V£, and the 
inequality (35 1 is from V 2 £ ^ — al derived above. Hence we proved the result. ■ 



Appendix D 
Proof of Lemma|4] 

Proof: We first consider a time division MWQ policy. At each time slot, only the link is selected 
for transmission and the policy is given in the following (6j. 

1) Find a link I such that 

1 = arg^max^{<#log(l + \h\ 2 pi) - Vpi} (36) 

2) Power allocation: the power p is allocated according to 



Pi 



3i l ) I - I 

V Jp t (37) 

otherwise 



where the projection yields pt = max{0, mm{qi/V — 1/ |/i/| 2 , 2 LAmax / /i 2 ,}} . 
3) Rate allocation: the rate is allocated according to 



lo S [l+(M^-l) ) 1 = 1 



V 



in - { V ^ V '»/ (38i 
otherwise 



Note that the above policy is the solution of the following optimization problem, 

maximize Y, q t log(l + \h\ 2 p{) - V YjPi (39) 

subject to only one link is activated. 
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As a result, the optimum queue-weighted sum transmission rate for the time division policy is 



L 

1=1 



■To£ 



1 + 



m 

V 



V 



> q m log 1 + 



V 



q m mm log H 



+ l oe M\ 



|q|| |^m| < V 
|q|| |/i m | 2 > V 



> 



|q|| min |log (^||q||) , LA max + log jf^j 



|q||£ 
|q||5, 



nun ^ y 

n un y 



(40) 



where q n 



|q|| stands for the queue that has the largest backlog (i.e., m = argmax; {qi}). The 



optimal utility for the time division policy is then given by (40 1 for HqHS'min > V where Pt 
2~2pi = P[ is the total power. 



Since, with the same objective, the optimization domain of the time division MWQ problem ( [39] ) 
is just a subset of that of the original MWQ problem in Q, the MWQ problem Q yields a utility 
U* = Y^qil^ ~ VJ^Pi — U. To evaluate the queue-weighted utility J^Qi^i' we consider the 
following two cases. 

Case 1: When J^Pi ^ Pt = YliPu it is obvious that, for ||q||S mm > V, 



, LA max + log , 



mm 




^2qii4 > ^2,<im > l|q||mm<j log v v 

Case 2: When 2~2p*i < Pu we l et V = ^p^V < V- Note that decreasing the tradeoff parameter V 
will increase the power allocation and hence increase the queue- weighted utility 2^2 HP* ■ Specifically, 
the optimal utility becomes 



U* 



- y j>? = XV? - v ' p * * X*^ - v ' Pt ^ X^i - v YJi 



as 



Pi 



l 



^ > Pi = Pt 1 = 1 

Pi 



otherwise 



where and p ; are the solutions to the time division MWQ problem (39). Hence J^Qif 1 ! ^ Yl^lV-i — 
l|q|| log (%P||q||) > ||q||min|log(^||q||) , LA max + log jf^ik for Hq^m > V. 
Combining the above two cases, we prove the inequality (20 1. 

In addition, as L||q|| > ^=i«M* > l|q|| min |log ||q||) , LA max + log jf^jf | , we have 
||/x*|| > iminjlog (^p||q||) , LA max + log jf^jf }, for ||q||5 miD > V. Similarly, we can get || < 
log (^y^llqll)- Hence we prove inequality (21 
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Appendix E 
Proof of Lemma[5] 

Proof: According to |7j) in Lemma [TJ ^(k) = log(Pfe(p)) under some permutation iv, where 

Pfc(p) 



i + Etx 


K(i) 


2 

Ptt(z) 


i + Eti 1 


K(i) 


2 

Ptt(i) 



i 1 2 

Notice that pfc(p) is a ratio of two polynomials. In addition, the coefficients are bounded by 

S mm and S max . Hence pk(p) is Lipschitz continuous, i.e., there exists < fa < oo depending on 
S min and S max such that 

\\Pk(p) ~ P(P*)\\ < fa\\P - P*|| = fa\\Pe\\- 

Therefore, as pk(.) > 1, assuming pk{p) > p(p*), we have 
HA«-{fc)(p) — A7r(fc)(P*)ll = l°g(Pfc(p)) -log(Pfc(P*)) 

n f, . Pfc(p) - Pfc(P*)\ ^, A , /%||Pe||\ n . R || |n 

= log 1+ 7-^ < log 1 + 7—r < log (1 + AfeHPell) 



Pk{P ) J \ Pk\P* 

Similarly, when pk(p) < p(p*), we have 

IIAw(fc)(p)-/*w(*)(P*)ll = log(pjfe(p*)) - log(pjfe(p)) 

, (-, , Pfc(P*) - Pfc(p)\ . , A , /3fe||Pen . , , , , n 

= log 1+ r~\ < log 1 + 1-^ < log (1 + fa\\Pe 



Pk{p) J V Pk(p) 

Hence ||/i(p) - /x(p*)|| < log (1 + /3||p e ||), where f3 = max k ={i,...,L} fa- Using the triangular 
inequality, we obtain 

\\fi(p*)\\ -log(l + /3|| Pe ||) < ||A(p)|| < ||A(P*)II +log(l + /3|| Pe ||) 
that leads to the result. ■ 

Appendix F 
Proof of Lemma[6] 

Proof: From Lemma [T] and [5] we have 

> J>[^-log(l + /?|| Pe ||)] + 

> E^-E* lo g( 1+ ^iip-ii) 



> ||q|| min <{ log ( -^||q|M , L\ max + log }> - L||q|| log (1 + /3||p e 



From the optimality condition [26| for a convex problem, we also have p^ /(p e ; h, q) < for all 



p e . According to the proof of Lemma [4] in Appendix |Dj two cases for q(t) should be considered. 
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Case 1: ||q||5 m i n > V. The stochastic Lyapunov drift (18 1 can be written as 

V 



LV(z) < -2 K a||p e || 2 + 2 7g ||p e ||log -^||q|| + 2 7? ||p e || log (1 + /3||p e 



+7/i«A||p e ||||h|| - aA||h|| 2 + 27 g A ma x||p e || + 2L||q|| log(l + f3\\p 

5rnin 1 1 1 1 \ T \ ,i S ] 

— ||q|| I ,LA max + log — 



2||q|| min<j log ( -^||q|| ) , LA max + log n E T | \ + 2L||q||A max + C (41) 



where 



2(A^ ^^3+^1+5^(20, + ^) < ^2^(1 + 7^) + L 7 2 + M 

' i=i i=i i=i 



< L(2a j4 (l + 7 2)+ 7 2 + A r 



4 C 



To find the upper bound of the R.H.S. of (41 1, we divide it into 2 parts as follows. 



J x = - K a||p e || 2 + 27qA max ||p e || - ||q|| min jlog ^^^Hqll^ , £A max + log 

+£A max ||q|| + 2 7 ^OA||p e ||||h|| - 2a j4 ||h|| 2 + C, 
I 2 = - K a||p e || 2 + 27 g ||p e || log f-^||q||^ + 27g||p e || log (1 + /3||p e ||) + LA max ||q|| 

+2L||q|| log(l + /3||p e ||) - ||q|| min jlog ( %^-||q|| J , £A max + lof " 



V ) \h \ 2 

(1) With some calculations, it is not difficult to show that Ii < -llhll + 7 ' A "* ax + _^2 LAma - -1 + 
5 — n t — m m + C, for k > — max |7 2 /37 }. 

(2) Denote gi(S min , S max ) = max {MM \ } {I 2 + (||p e || + ||q||)}. We can easily find that #1 is 
bounded above for all S mm and Smax in the domaiij^] Note that an upper bound expression for gi 
is always obtainable, since it is only a simple bivariate programming problem. Therefore, we obtain 

h < ~ (||Pe|| + ||q||) + 9l{S m in, S max ). 

As a result, we have 

2 \ 2 

LV < -(|| Pe || + || q || + ||h||) + ^™ + -^2 LA -- 1 

KOL '-'mill 

+ 77 Fi n 777 77 + 9l(S mm , S max ) + C. 
8a A [l-%a A /{ka)\ 

Case 2: ||q||S , m in < V. Here we have ||q|| < From the property in Appendix |d| the stochastic 



9 To show a real valued function f(x,y) is bounded above, we start from a point (xo,yo) in the domain and proceed to 
show that, by substituting with y = xq + fi{y — yo), f(x, y(x; /3)) is bounded above uniformly for every /3 £ R. It can be 
verified that f(x,y(x;/3)) satisfies this condition in our case. 
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Lyapunov drift ( 1 8 1 can be written as 

LV(z) < -2Ka||p e || 2 + 27 9 ||p e || log {S max S min ) + 27 9 ||p e || log (1 + /3||p e ||) + 7fr.a J 4||p e || ||h| 



< 



where 



V 



V 



+27 5 A max ||p e || - a A \\h\\ + - — log (1 + /3\\p e \\) + - — LA max + C 



5, 



< - (||p e || + ||h||) +0j(S J nin,S n »x) + 



1 



Sr, 



7 2 A 2 
iq ''max 



^min max 8a A [l - ila A /(ka)) 



+ C 



7 2 A 2 

|p e || + ||h||) + Jo + + — LX max 



V 

KOI Srmn 



8a A [l - %a A /(ka) 



+ C- ||q|| + 



92{S mm , S n 



max{-Ka||p e || 2 + 27 9 ||p e || log (S^A 



V 



+2 lq \\ Pe \\ log (1 + f3\\p e \\) + — log (1 + f3\\ Pe \\)}. 



Therefore, we have [since C = L(2a A (l + 7 2 ) + 7 2 + A max )] 



+ 



7 2 A 2 V 

Iq 'max ' 



+ 



2 LA max -l + 



LV(z) < -(||p e || + ||q|| + ||h||) + L(2a A (l + 7^)+7 2 + A max ) 

1 



Ka ' S min ' 8a A [l - ~fla A /(ka)] 

where g(S m ini S m nx) = m & x {fi'l('Smin> $max)>5l($miri) 'S'max)}- 



Appendix G 
Proof of Theorem @] 

Proof: Consider the virtual error dynamic system 

dp e = nVpC (£(p), P; h, q) dt + (<p q (p; .) - q (p*;.)) dq + Re ((p h (p; .)dh - h (p*; .)dh) 
kV p C (/i(p), p; h, q) - n(t) (if q (p; .) - q (p*;.)) - ^Re [(<p h (p; .) - <p h (p*;.)) Ah] 
+ (0 q (p; .) - <p q (p*; .)) dN(t) + Re [(0 & (p; .) - ^(p*; .)) A>dW(t) 



dt 
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Taking the Lyapunov function as V(p e ) = |p^p e - The Lyapunov drift is defined as LV{p e ) = 
lim^o j [V (p e (t + 5)) \p e (t)] - V (p e (t))}. Note that E [hi] = 0. The drift can be derived into 



LV( Pe 



pT/(p e ; ■) - pM*) (^(p; ■) - ^(p*; ■)) + £ A ^ (4°(p; " £?(p*; ■)) 

(<£ g (p; .) - ^(p*; .)) T (<£ 9 (p; ■) - 9 (p*; .))] 



1 

+ 2 tt 
1 

H — tr 
2 



A") (^h(p;.) - ^ft(p*;0) (^h(p;«) -^(p*;-))^ 5 

^1 

2 91 



2 ii2 1 1 2 2 1 1 1 1 2 2 1 1 1 1 2 

< — Ka||p e || + Mmax^gllPell + A max LLq || p e || +-L (? ||p e || +-O^L h ||p e || 

= -^llPell 2 



where p = -ra + fJ- max L q + X mSLX LL q + ^L^ + \ a AL\ > 0. Hence from the asymptotic stochastic 
stability results given in ]25[ we have proven the theorem. ■ 
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